benchmarks: Implement benchmarks for streamable #10388

xdustinface · 2022-02-23T04:03:32Z

Implements some benchmarks for streamable object creation, byte conversions and json conversions for the classes FullBlock and BenchmarkClass. The BenchmarkClass can probably still be improved so im open for suggestions :) The benchmark collects iterations over a specific time window and this can be done multiple runs to get an average result. See the parameter below:

Usage

python benchmarks/streamable.py --help
Usage: streamable.py [OPTIONS]

Options:
  -d, --data [all|benchmark|full_block]
  -m, --mode [all|creation|to_bytes|from_bytes|to_json|from_json]
  -r, --runs INTEGER              Number of benchmark runs to average results
  -t, --ms INTEGER                Milliseconds per run
  --live / --no-live              Print live results (slower)
  --help                          Show this message and exit.

Example run

Run all benchmarks with the class: BenchmarkClass
Run all benchmarks with the class: BenchmarkClass
runs       | ms/run     | ns/iteration | mode       | avg iterations | stdev iterations
100        | 47         | 18           | creation   |           2536 |         22.81
100        | 50         | 462          | to_bytes   |            108 |          0.69
100        | 49         | 275          | from_bytes |            181 |          3.92
100        | 50         | 927          | to_json    |             54 |          1.38
100        | 50         | 1089         | from_json  |             46 |          0.89

Run all benchmarks with the class: FullBlock
runs       | ms/run     | ns/iteration | mode       | avg iterations | stdev iterations
100        | 48         | 23           | creation   |           2077 |         63.08
100        | 49         | 114          | to_bytes   |            434 |         10.39
100        | 51         | 2923         | from_bytes |             17 |          0.42
100        | 50         | 422          | to_json    |            118 |          2.01
100        | 51         | 2974         | from_json  |             17 |          0.45

Benchmark classes

@dataclass(frozen=True)
@streamable
class BenchmarkInner(Streamable):
    a: str


@dataclass(frozen=True)
@streamable
class BenchmarkMiddle(Streamable):
    a: uint64
    b: List[bytes32]
    c: Tuple[str, bool, uint8, List[bytes]]
    d: Tuple[BenchmarkInner, BenchmarkInner]
    e: BenchmarkInner


@dataclass(frozen=True)
@streamable
class BenchmarkClass(Streamable):
    a: Optional[BenchmarkMiddle]
    b: Optional[BenchmarkMiddle]
    c: BenchmarkMiddle
    d: List[BenchmarkMiddle]
    e: Tuple[BenchmarkMiddle, BenchmarkMiddle, BenchmarkMiddle]

mariano54 · 2022-02-23T13:30:54Z

Very cool. I suggest naming the file bench_streamable so when we search for filenames we don't get duplicates with ide. Also it would be nice to have the avg ms time for each iteration

xdustinface · 2022-02-23T13:46:49Z

Very cool. I suggest naming the file bench_streamable so when we search for filenames we don't get duplicates with ide.

I don't care that much and i can rename if you really want it but isn't that what folders and packages are about? We also have util.py in several places and i mean, even with the bench prefix it gets listed as you can see with the test_streamable :)

Also it would be nice to have the avg ms time for each iteration

Not sure if i understand what you mean exactly. What it currently prints as result is "sum of runs per data and mode" / runs.

chia-blockchain/benchmarks/streamable.py

Line 187 in 58240e2

average_duration: int = int(sum(results[current_data][current_mode]) / runs)

Can you explain what you would like to see?

mariano54 · 2022-02-23T14:17:08Z

It's personal preference, I like to be able to just press 4-5 keys on the keyboard and navigate to a file deterministically without having to check which folder it's in, etc. No big deal though

mariano54 · 2022-02-23T14:20:04Z

The result is always 1 second. That seems wrong, it should be the time taken for 1 iteration right?

xdustinface · 2022-02-23T16:00:17Z

The result is always 1 second. That seems wrong

That was by purpose actually. I trimmed the number of iterations per test separetly so that they all have about the same runtime.

See 06356bc for another approach which is to "collect the iterations for a given time" instead of "measuring the time for a given number of iterations". What do you think? I adjusted the initial desctiption but still here is an example run:

Run all benchmarks with the class: BenchmarkClass
runs       | ms/run     | mode       | avg iterations
50         | 50         | creation   |          2618
50         | 50         | to_bytes   |           110
50         | 50         | from_bytes |           175
50         | 50         | to_json    |            53
50         | 50         | from_json  |            45

Run all benchmarks with the class: FullBlock
runs       | ms/run     | mode       | avg iterations
50         | 50         | creation   |          2140
50         | 50         | to_bytes   |           449
50         | 50         | from_bytes |            18
50         | 50         | to_json    |           111
50         | 50         | from_json  |            16

xdustinface · 2022-02-23T18:08:16Z

Added std deviation field, see:

Run all benchmarks with the class: BenchmarkClass
runs       | ms/run     | mode       | avg iterations | std deviation
50         | 50         | creation   |           2593 |         27.46
50         | 50         | to_bytes   |            110 |          1.02
50         | 50         | from_bytes |            175 |          4.11
50         | 50         | to_json    |             53 |          1.38
50         | 50         | from_json  |             45 |           0.6

Run all benchmarks with the class: FullBlock
runs       | ms/run     | mode       | avg iterations | std deviation
50         | 50         | creation   |           2126 |         25.88
50         | 50         | to_bytes   |            439 |          9.65
50         | 50         | from_bytes |             17 |          0.63
50         | 50         | to_json    |            115 |          1.59
50         | 50         | from_json  |             16 |          0.49

mariano54 · 2022-02-23T19:14:08Z

I'ts nice because it means you don't have to wait a long time for the slow operations. I prefer to additionally look at the ms (or nanoseconds) taken per run as a way to visualize the speed of something, as opposed to trying to increase the number of iterations.

benchmarks/streamable.py

xdustinface · 2022-02-23T22:11:41Z

@mariano54 Added ns/iteration, example run:

Run all benchmarks with the class: BenchmarkClass
runs       | ms/run     | ns/iteration | mode       | avg iterations | stdev iterations
100        | 47         | 18           | creation   |           2536 |         22.81
100        | 50         | 462          | to_bytes   |            108 |          0.69
100        | 49         | 275          | from_bytes |            181 |          3.92
100        | 50         | 927          | to_json    |             54 |          1.38
100        | 50         | 1089         | from_json  |             46 |          0.89

Run all benchmarks with the class: FullBlock
runs       | ms/run     | ns/iteration | mode       | avg iterations | stdev iterations
100        | 48         | 23           | creation   |           2077 |         63.08
100        | 49         | 114          | to_bytes   |            434 |         10.39
100        | 51         | 2923         | from_bytes |             17 |          0.42
100        | 50         | 422          | to_json    |            118 |          2.01
100        | 51         | 2974         | from_json  |             17 |          0.45

xdustinface · 2022-02-24T00:54:03Z

Okay improved the logging a bit. I think thats it from me now, any other requests? Example:

runs: 100, ms/run: 50, benchmarks: all, data: BenchmarkClass
mode       | µs/iteration | avg iterations | stdev iterations %
creation   | 19           |           2518 |                0.4
to_bytes   | 476          |            105 |               1.43
from_bytes | 290          |            172 |               0.75
to_json    | 967          |             52 |               1.92
from_json  | 1130         |             44 |               0.93

runs: 100, ms/run: 50, benchmarks: all, data: FullBlock
mode       | µs/iteration | avg iterations | stdev iterations %
creation   | 23           |           2056 |               1.18
to_bytes   | 114          |            433 |                0.9
from_bytes | 2846         |             17 |               0.58
to_json    | 435          |            115 |               1.06
from_json  | 3165         |             16 |               2.56

xdustinface · 2022-02-24T03:11:14Z

I think thats it from me now

wasn't, but now.. :

runs: 100, ms/run: 50, benchmarks: all, data: BenchmarkClass
mode       | µs/iteration | stdev µs/iteration % | avg iterations/run | stdev iterations/run %
creation   | 18.24        |                 6.85 |               2465 |                   0.58
to_bytes   | 455.4        |                 1.66 |                109 |                   1.06
from_bytes | 275.55       |                 2.01 |                180 |                    1.0
to_json    | 946.72       |                 3.76 |                 53 |                   2.25
from_json  | 1086.42      |                 2.47 |                 46 |                   1.54

runs: 100, ms/run: 50, benchmarks: all, data: FullBlock
mode       | µs/iteration | stdev µs/iteration % | avg iterations/run | stdev iterations/run %
creation   | 22.83        |                 6.78 |               2016 |                   0.97
to_bytes   | 112.65       |                 4.93 |                437 |                   1.15
from_bytes | 2775.47      |                 2.04 |                 18 |                   2.63
to_json    | 424.53       |                 4.76 |                117 |                   1.11
from_json  | 3140.45      |                 4.96 |                 16 |                   3.19

mariano54 · 2022-02-24T03:37:31Z

Looks really good now.

xdustinface requested review from arvidn and mariano54 February 23, 2022 04:03

benchmarks: Implement benchmarks for streamable

58240e2

xdustinface force-pushed the pr-benchmark-streamable branch from b292fc4 to 58240e2 Compare February 23, 2022 13:08

benchmarks: Collect iterations per time instead of time per iterations

06356bc

benchmarks: Add standard deviation to streamable benchs

54341b4

Nikolaj-K reviewed Feb 23, 2022

View reviewed changes

benchmarks/streamable.py Outdated Show resolved Hide resolved

benchmarks/streamable.py Outdated Show resolved Hide resolved

altendky reviewed Feb 23, 2022

View reviewed changes

benchmarks/streamable.py Outdated Show resolved Hide resolved

xdustinface added 2 commits February 23, 2022 22:55

benchmarks: Add ns/iteration to streamable benchs

e4fbf73

benchmarks: Move object creation out or the runs loop

2c931a9

xdustinface added 3 commits February 24, 2022 00:13

benchmarks: Use click.Choice for --data and --mode

ecb1bf5

benchmarks: Its µ

8cabece

benchmarks: Improve logging

ae28873

xdustinface added 3 commits February 24, 2022 02:58

benchmarks: Drop unused code

9b958d5

benchmarks: Use process_time as clock

a0520e8

benchmarks: Add stdev us/iterations % + more precission

70669cb

xdustinface force-pushed the pr-benchmark-streamable branch from a32c2bb to 05686a5 Compare February 24, 2022 03:03

benchmarks: Add --live/--no-live option to enable live results

0ad521e

xdustinface force-pushed the pr-benchmark-streamable branch from 05686a5 to 0ad521e Compare February 24, 2022 03:09

mariano54 approved these changes Feb 24, 2022

View reviewed changes

wjblanke merged commit cdf48f7 into Chia-Network:main Feb 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

benchmarks: Implement benchmarks for streamable #10388

benchmarks: Implement benchmarks for streamable #10388

xdustinface commented Feb 23, 2022 •

edited

Loading

mariano54 commented Feb 23, 2022

xdustinface commented Feb 23, 2022

mariano54 commented Feb 23, 2022

mariano54 commented Feb 23, 2022

xdustinface commented Feb 23, 2022

xdustinface commented Feb 23, 2022

mariano54 commented Feb 23, 2022

xdustinface commented Feb 23, 2022

xdustinface commented Feb 24, 2022

xdustinface commented Feb 24, 2022

mariano54 commented Feb 24, 2022 •

edited

Loading

benchmarks: Implement benchmarks for streamable #10388

benchmarks: Implement benchmarks for streamable #10388

Conversation

xdustinface commented Feb 23, 2022 • edited Loading

Usage

Example run

Benchmark classes

mariano54 commented Feb 23, 2022

xdustinface commented Feb 23, 2022

mariano54 commented Feb 23, 2022

mariano54 commented Feb 23, 2022

xdustinface commented Feb 23, 2022

xdustinface commented Feb 23, 2022

mariano54 commented Feb 23, 2022

xdustinface commented Feb 23, 2022

xdustinface commented Feb 24, 2022

xdustinface commented Feb 24, 2022

mariano54 commented Feb 24, 2022 • edited Loading

xdustinface commented Feb 23, 2022 •

edited

Loading

mariano54 commented Feb 24, 2022 •

edited

Loading